{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "对单词和字符进行one-hot编码"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [],
   "source": [
    "import numpy as np\n",
    "\n",
    "samples = ['The cat sat on the mat.', 'The dog ate my homework.']\n",
    "\n",
    "token_index = {}\n",
    "for sample in samples:\n",
    "    for word in sample.split():\n",
    "        if word not in token_index:\n",
    "            token_index[word] = len(token_index) + 1\n",
    "\n",
    "max_length = 10\n",
    "\n",
    "results = np.zeros(shape=(len(samples), max_length, max(token_index.values()) + 1))\n",
    "for i, sample in enumerate(samples):\n",
    "    for j, word in list(enumerate(sample.split()))[:max_length]:\n",
    "            index = token_index.get(word)\n",
    "            results[i, j, index] = 1"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "字母级的one_hot编码略，手动方式太麻烦，用 keras 内置的函数："
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Using TensorFlow backend.\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Found 9 unique tokens.\n"
     ]
    }
   ],
   "source": [
    "from keras.preprocessing.text import Tokenizer\n",
    "\n",
    "samples = ['The cat sat on the mat.', 'The dog ate my homework.']\n",
    "tokenizer = Tokenizer(num_words=1000)\n",
    "tokenizer.fit_on_texts(samples)\n",
    "sequences = tokenizer.texts_to_sequences(samples)\n",
    "one_hot_results = tokenizer.texts_to_matrix(samples, mode='binary')\n",
    "word_index = tokenizer.word_index\n",
    "print('Found %s unique tokens.' % len(word_index))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**One-hot 散列技巧"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [],
   "source": [
    "samples = ['The cat sat on the mat.', 'The dog ate my homework.']\n",
    "dimensionality = 1000\n",
    "max_length = 10\n",
    "\n",
    "results = np.zeros((len(samples), max_length, dimensionality))\n",
    "for i, sample in enumerate(samples):\n",
    "    for j, word in list(enumerate(sample.split()))[:max_length]:\n",
    "        index = abs(hash(word)) % dimensionality\n",
    "        results[i, j, index] = 1."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 词嵌入\n",
    "\n",
    "使用密集的词向量编码，但是向量是从数据中学习到的。\n",
    "\n",
    " * 完成主任务的同时学习词嵌入，一开始的时候是随机词向量，逐渐学习到权重\n",
    " * 在任务上预计算好词嵌入，加载到模型中。（预训练词嵌入）\n",
    " \n",
    "**词嵌入**：把一个词转换成一个向量，不同的向量之间的空间关系应该包含了词语之间的关系:\n",
    " * 2个词之间的距离代表了语义的相似程度\n",
    " * 词向量的*方向*也具有一定的含义\n",
    " \n",
    "比如：🐺->🐶 和 🐯->🐈 在词空间上的向量差应该是相近的：代表了他们从野生到家养的关系。\n",
    "\n",
    "再比如：$V_{king} + V_{female} = V_{queen}$ &nbsp;和 $V_{king} + V_{plural} = V_{kings}$\n",
    "\n",
    "  虽然可以把大量的词在词向量空间内找到位置，但是目前不存在一个完美映射人类语言的词嵌入空间。\n",
    "\n",
    "  而且人类的语言在不同的领域，不同的语种对应的词空间可能也不同。\n",
    "\n",
    "  因此更好的办法是：每个学习任务使用一个预训练好的嵌入空间。反向传播让这个过程变的简单。Keras中仅需要学习 Embedding 层的权重即可。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [],
   "source": [
    "from keras.layers import Embedding\n",
    "\n",
    "embedding_layer = Embedding(1000, 64)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    " * 1000: 标记的个数\n",
    " * 64: 嵌入的维度"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "现在对IMDB数据集上的评论进行情感分类"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [],
   "source": [
    "from keras.datasets import imdb\n",
    "from keras import preprocessing\n",
    "\n",
    "max_features = 10000\n",
    "maxlen = 20\n",
    "\n",
    "(x_train, y_train), (x_test, y_test) = imdb.load_data(num_words=max_features)\n",
    "\n",
    "x_train = preprocessing.sequence.pad_sequences(x_train, maxlen=maxlen)\n",
    "x_test = preprocessing.sequence.pad_sequences(x_test, maxlen=maxlen)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "WARNING:tensorflow:From /usr/local/lib/python3.7/site-packages/keras/backend/tensorflow_backend.py:488: The name tf.placeholder is deprecated. Please use tf.compat.v1.placeholder instead.\n",
      "\n",
      "WARNING:tensorflow:From /usr/local/lib/python3.7/site-packages/keras/backend/tensorflow_backend.py:3626: The name tf.random_uniform is deprecated. Please use tf.random.uniform instead.\n",
      "\n",
      "WARNING:tensorflow:From /usr/local/lib/python3.7/site-packages/keras/backend/tensorflow_backend.py:1255: calling reduce_prod_v1 (from tensorflow.python.ops.math_ops) with keep_dims is deprecated and will be removed in a future version.\n",
      "Instructions for updating:\n",
      "keep_dims is deprecated, use keepdims instead\n",
      "WARNING:tensorflow:From /usr/local/lib/python3.7/site-packages/keras/optimizers.py:711: The name tf.train.Optimizer is deprecated. Please use tf.compat.v1.train.Optimizer instead.\n",
      "\n",
      "WARNING:tensorflow:From /usr/local/lib/python3.7/site-packages/keras/backend/tensorflow_backend.py:2921: The name tf.log is deprecated. Please use tf.math.log instead.\n",
      "\n",
      "WARNING:tensorflow:From /usr/local/lib/python3.7/site-packages/tensorflow_core/python/ops/nn_impl.py:183: where (from tensorflow.python.ops.array_ops) is deprecated and will be removed in a future version.\n",
      "Instructions for updating:\n",
      "Use tf.where in 2.0, which has the same broadcast rule as np.where\n",
      "_________________________________________________________________\n",
      "Layer (type)                 Output Shape              Param #   \n",
      "=================================================================\n",
      "embedding_3 (Embedding)      (None, 20, 8)             80000     \n",
      "_________________________________________________________________\n",
      "flatten_1 (Flatten)          (None, 160)               0         \n",
      "_________________________________________________________________\n",
      "dense_1 (Dense)              (None, 1)                 161       \n",
      "=================================================================\n",
      "Total params: 80,161\n",
      "Trainable params: 80,161\n",
      "Non-trainable params: 0\n",
      "_________________________________________________________________\n",
      "WARNING:tensorflow:From /usr/local/lib/python3.7/site-packages/keras/backend/tensorflow_backend.py:671: calling Constant.__init__ (from tensorflow.python.ops.init_ops) with dtype is deprecated and will be removed in a future version.\n",
      "Instructions for updating:\n",
      "Call initializer instance with the dtype argument instead of passing it to the constructor\n",
      "WARNING:tensorflow:From /usr/local/lib/python3.7/site-packages/keras/backend/tensorflow_backend.py:949: The name tf.assign_add is deprecated. Please use tf.compat.v1.assign_add instead.\n",
      "\n",
      "WARNING:tensorflow:From /usr/local/lib/python3.7/site-packages/keras/backend/tensorflow_backend.py:936: The name tf.assign is deprecated. Please use tf.compat.v1.assign instead.\n",
      "\n",
      "WARNING:tensorflow:From /usr/local/lib/python3.7/site-packages/keras/backend/tensorflow_backend.py:2353: The name tf.Session is deprecated. Please use tf.compat.v1.Session instead.\n",
      "\n",
      "Train on 20000 samples, validate on 5000 samples\n",
      "Epoch 1/10\n",
      "WARNING:tensorflow:From /usr/local/lib/python3.7/site-packages/keras/backend/tensorflow_backend.py:158: The name tf.get_default_session is deprecated. Please use tf.compat.v1.get_default_session instead.\n",
      "\n",
      "WARNING:tensorflow:From /usr/local/lib/python3.7/site-packages/keras/backend/tensorflow_backend.py:163: The name tf.ConfigProto is deprecated. Please use tf.compat.v1.ConfigProto instead.\n",
      "\n",
      "WARNING:tensorflow:From /usr/local/lib/python3.7/site-packages/keras/backend/tensorflow_backend.py:172: The name tf.global_variables is deprecated. Please use tf.compat.v1.global_variables instead.\n",
      "\n",
      "WARNING:tensorflow:From /usr/local/lib/python3.7/site-packages/keras/backend/tensorflow_backend.py:180: The name tf.is_variable_initialized is deprecated. Please use tf.compat.v1.is_variable_initialized instead.\n",
      "\n",
      "WARNING:tensorflow:From /usr/local/lib/python3.7/site-packages/keras/backend/tensorflow_backend.py:187: The name tf.variables_initializer is deprecated. Please use tf.compat.v1.variables_initializer instead.\n",
      "\n",
      "20000/20000 [==============================] - 1s 56us/step - loss: 0.6704 - acc: 0.6171 - val_loss: 0.6227 - val_acc: 0.6948\n",
      "Epoch 2/10\n",
      "20000/20000 [==============================] - 1s 52us/step - loss: 0.5476 - acc: 0.7452 - val_loss: 0.5239 - val_acc: 0.7452\n",
      "Epoch 3/10\n",
      "20000/20000 [==============================] - 1s 45us/step - loss: 0.4655 - acc: 0.7823 - val_loss: 0.4943 - val_acc: 0.7568\n",
      "Epoch 4/10\n",
      "20000/20000 [==============================] - 1s 45us/step - loss: 0.4242 - acc: 0.8079 - val_loss: 0.4860 - val_acc: 0.7622\n",
      "Epoch 5/10\n",
      "20000/20000 [==============================] - 1s 40us/step - loss: 0.3969 - acc: 0.8216 - val_loss: 0.4881 - val_acc: 0.7602\n",
      "Epoch 6/10\n",
      "20000/20000 [==============================] - 1s 40us/step - loss: 0.3749 - acc: 0.8356 - val_loss: 0.4923 - val_acc: 0.7596\n",
      "Epoch 7/10\n",
      "20000/20000 [==============================] - 1s 41us/step - loss: 0.3559 - acc: 0.8454 - val_loss: 0.4976 - val_acc: 0.7582\n",
      "Epoch 8/10\n",
      "20000/20000 [==============================] - 1s 40us/step - loss: 0.3382 - acc: 0.8547 - val_loss: 0.5044 - val_acc: 0.7560\n",
      "Epoch 9/10\n",
      "20000/20000 [==============================] - 1s 45us/step - loss: 0.3213 - acc: 0.8637 - val_loss: 0.5115 - val_acc: 0.7548\n",
      "Epoch 10/10\n",
      "20000/20000 [==============================] - 1s 53us/step - loss: 0.3048 - acc: 0.8727 - val_loss: 0.5211 - val_acc: 0.7516\n"
     ]
    }
   ],
   "source": [
    "from keras.models import Sequential\n",
    "from keras.layers import Flatten, Dense, Embedding\n",
    "\n",
    "model = Sequential()\n",
    "model.add(Embedding(10000, 8, input_length=maxlen))\n",
    "model.add(Flatten())\n",
    "model.add(Dense(1, activation='sigmoid'))\n",
    "model.compile(optimizer='rmsprop', loss='binary_crossentropy', metrics=['acc'])\n",
    "model.summary()\n",
    "\n",
    "history = model.fit(x_train, y_train, epochs=10, batch_size=32, validation_split=0.2)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "最终得到的验证精度大约76%。在最终的输出层上只使用了一个 Dense 层，把整个序列中的token都独立处理没有考虑到词语序列的影响。因此后续需要添加卷积层，把整个序列当做一个整体考虑。"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "从最原始的数据开始，一步步计算得到自己的词向量空间。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {},
   "outputs": [],
   "source": [
    "import os\n",
    "\n",
    "imdb_dir = '/Users/haoxingxiao/academic/data/aclImdb'\n",
    "train_dir = os.path.join(imdb_dir, 'train')\n",
    "\n",
    "labels = []\n",
    "texts = []\n",
    "\n",
    "for label_type in ['neg', 'pos']:\n",
    "    dir_name = os.path.join(train_dir, label_type)\n",
    "    for fname in os.listdir(dir_name):\n",
    "        if fname[-4:] == '.txt':\n",
    "            f = open(os.path.join(dir_name, fname))\n",
    "            texts.append(f.read())\n",
    "            f.close()\n",
    "            if label_type == 'neg':\n",
    "                labels.append(0)\n",
    "            else:\n",
    "                labels.append(1)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Found 88582 unique tokens.\n",
      "Shape of data tensor: (25000, 100)\n",
      "Shape of label tensor: (25000,)\n"
     ]
    }
   ],
   "source": [
    "from keras.preprocessing.text import Tokenizer\n",
    "from keras.preprocessing.sequence import pad_sequences\n",
    "import numpy as np\n",
    "\n",
    "maxlen = 100\n",
    "training_samples = 200\n",
    "validation_samples = 10000\n",
    "max_words = 10000\n",
    "\n",
    "tokenizer = Tokenizer(num_words=max_words)\n",
    "tokenizer.fit_on_texts(texts)\n",
    "sequences = tokenizer.texts_to_sequences(texts)\n",
    "\n",
    "word_index = tokenizer.word_index\n",
    "print('Found %s unique tokens.' % len(word_index))\n",
    "\n",
    "data = pad_sequences(sequences, maxlen=maxlen)\n",
    "\n",
    "labels = np.asarray(labels)\n",
    "print('Shape of data tensor:', data.shape)\n",
    "print('Shape of label tensor:', labels.shape)\n",
    "\n",
    "indices = np.arange(data.shape[0])\n",
    "np.random.shuffle(indices)\n",
    "data = data[indices]\n",
    "labels = labels[indices]\n",
    "\n",
    "x_train = data[:training_samples]\n",
    "y_train = labels[:training_samples]\n",
    "x_val = data[training_samples: training_samples + validation_samples]\n",
    "y_val = labels[training_samples: training_samples + validation_samples]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 23,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Found 400000 word vectors\n"
     ]
    }
   ],
   "source": [
    "glove_dir = '/Users/haoxingxiao/academic/data/glove.6B'\n",
    "embeddings_index = {}\n",
    "f = open(os.path.join(glove_dir, 'glove.6B.100d.txt'))\n",
    "for line in f:\n",
    "    values = line.split()\n",
    "    word = values[0]\n",
    "    coefs = np.asarray(values[1:], dtype='float32')\n",
    "    embeddings_index[word] = coefs\n",
    "f.close()\n",
    "\n",
    "print('Found %s word vectors' % len(embeddings_index))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 24,
   "metadata": {},
   "outputs": [],
   "source": [
    "embedding_dim = 100\n",
    "\n",
    "# 嵌入索引找不到的词，嵌入向量全为 0\n",
    "embedding_matrix = np.zeros((max_words, embedding_dim))\n",
    "for word, i in word_index.items():\n",
    "    if i < max_words:\n",
    "        embedding_vector = embeddings_index.get(word)\n",
    "        if embedding_vector is not None:\n",
    "            embedding_matrix[i] = embedding_vector"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 定义模型架构"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 26,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "_________________________________________________________________\n",
      "Layer (type)                 Output Shape              Param #   \n",
      "=================================================================\n",
      "embedding_5 (Embedding)      (None, 100, 100)          1000000   \n",
      "_________________________________________________________________\n",
      "flatten_3 (Flatten)          (None, 10000)             0         \n",
      "_________________________________________________________________\n",
      "dense_3 (Dense)              (None, 32)                320032    \n",
      "_________________________________________________________________\n",
      "dense_4 (Dense)              (None, 1)                 33        \n",
      "=================================================================\n",
      "Total params: 1,320,065\n",
      "Trainable params: 1,320,065\n",
      "Non-trainable params: 0\n",
      "_________________________________________________________________\n"
     ]
    }
   ],
   "source": [
    "from keras.models import Sequential\n",
    "from keras.layers import Embedding, Flatten, Dense\n",
    "\n",
    "model = Sequential()\n",
    "model.add(Embedding(max_words, embedding_dim, input_length=maxlen))\n",
    "model.add(Flatten())\n",
    "model.add(Dense(32, activation='relu'))\n",
    "model.add(Dense(1, activation='sigmoid'))\n",
    "model.summary()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 27,
   "metadata": {},
   "outputs": [],
   "source": [
    "# 预训练的词嵌入加载到Embedding层中\n",
    "model.layers[0].set_weights([embedding_matrix])\n",
    "model.layers[0].trainable=False"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 29,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Train on 200 samples, validate on 10000 samples\n",
      "Epoch 1/10\n",
      "200/200 [==============================] - 1s 3ms/step - loss: 2.0353 - acc: 0.5650 - val_loss: 0.7119 - val_acc: 0.5355\n",
      "Epoch 2/10\n",
      "200/200 [==============================] - 0s 1ms/step - loss: 0.7517 - acc: 0.6750 - val_loss: 0.9303 - val_acc: 0.4968\n",
      "Epoch 3/10\n",
      "200/200 [==============================] - 0s 2ms/step - loss: 0.2929 - acc: 0.9000 - val_loss: 1.1469 - val_acc: 0.5099\n",
      "Epoch 4/10\n",
      "200/200 [==============================] - 0s 1ms/step - loss: 0.2840 - acc: 0.8600 - val_loss: 0.7777 - val_acc: 0.5519\n",
      "Epoch 5/10\n",
      "200/200 [==============================] - 0s 1ms/step - loss: 0.3608 - acc: 0.8250 - val_loss: 0.7326 - val_acc: 0.5757\n",
      "Epoch 6/10\n",
      "200/200 [==============================] - 0s 1ms/step - loss: 0.0597 - acc: 0.9950 - val_loss: 0.8325 - val_acc: 0.5664\n",
      "Epoch 7/10\n",
      "200/200 [==============================] - 0s 1ms/step - loss: 0.0452 - acc: 1.0000 - val_loss: 0.7763 - val_acc: 0.5804\n",
      "Epoch 8/10\n",
      "200/200 [==============================] - 0s 1ms/step - loss: 0.0225 - acc: 1.0000 - val_loss: 0.8464 - val_acc: 0.5757\n",
      "Epoch 9/10\n",
      "200/200 [==============================] - 0s 1ms/step - loss: 0.0153 - acc: 1.0000 - val_loss: 0.8083 - val_acc: 0.5796\n",
      "Epoch 10/10\n",
      "200/200 [==============================] - 0s 1ms/step - loss: 0.0088 - acc: 1.0000 - val_loss: 0.8939 - val_acc: 0.5726\n"
     ]
    }
   ],
   "source": [
    "# 训练模型和评估\n",
    "model.compile(optimizer='rmsprop',\n",
    "             loss='binary_crossentropy',\n",
    "             metrics=['acc'])\n",
    "history = model.fit(x_train, y_train, epochs=10, batch_size=32, validation_data=(x_val, y_val))\n",
    "model.save_weights('pre_trained_glove_model.h5')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 30,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAXQAAAEICAYAAABPgw/pAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADh0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uMy4xLjIsIGh0dHA6Ly9tYXRwbG90bGliLm9yZy8li6FKAAAgAElEQVR4nO3de3hU5bn38e9NADGcETyBELSKBCQQs/GAJ0QQbQsb9bLQeMDulm5btbVufVHb2vpuaveuWymtrxXdnmoaarUeWrW0trZqtZbgAQXkIARMOMhBEUGEwP3+8awkk5DDJBkyk5Xf57rmmpm11qx1ZyX5zTPPWvMsc3dERKTt65DuAkREJDUU6CIiMaFAFxGJCQW6iEhMKNBFRGJCgS4iEhMK9Jgysywz+8TMBqZy2XQys8+Z2QE5z7b2us3sj2ZWeCDqMLPvmdkvmvt6kfoo0DNEFKiVt31m9mnC8zqDpSHuvtfdu7n72lQum6nM7Hkz+34d0y80s3Izy2rK+tx9grsXpaCuc8ystNa6/6+7/3tL193INt3MrjtQ25DMpEDPEFGgdnP3bsBa4IsJ0/YLFjPr2PpVZrSHgEvrmH4p8Ii7723letLpcmArcFlrb1h/l+mlQG8jzOw/zezXZlZsZtuBS8zsFDP7h5l9ZGbrzWyOmXWKlu8YtdJyouePRPOfM7PtZvaqmQ1u6rLR/PPMbLmZbTOzn5nZ381sej11J1Pj181spZl9aGZzEl6bZWZ3mtkWM1sFTGxgF/0WONzMTk14/SHA+cDD0fNJZvammX1sZmvN7HsN7O+XK3+mxuows6+a2dJoX71nZl+NpvcEfgcMTPi0dWj0u3ww4fVTzGxxtI/+YmZDEuaVmdl3zOztaH8Xm9lBDdTdHbgA+AaQa2Yja80/I/p9bDOz983s0mh6dvQzro3mvWhmB9X1CSOq6azocZP+LqPXnBB9otpqZhvM7AYz629mO82sV8Jyo6P5epNIlrvrlmE3oBQ4p9a0/wR2A18kvBEfDPwLcBLQETgaWA5cFS3fEXAgJ3r+CLAZKAA6Ab8mtFybuuyhwHZgcjTvO8AeYHo9P0syNT4F9ARyCC3Lc6L5VwGLgQHAIcCL4U+23v32APCLhOffBEoSnp8NDIv2X170M34hmve5xHUDL1f+TI3VEf1OjgYs2sanwIho3jlAaR2/ywejx0OBT6LXdQJuApYBnaL5ZcA/gMOjbS8HvtrAPrgiek0H4DngzoR5g6NtXRzt+77AyGjePcCfgSOALOC0qJ666i8Dzmrm32VPYCPwLeAgoAcwOpr3R+BrCdv5WWL9uiWRHekuQLc6fin1B/pfGnndfwC/iR7XFdKJYTcJeKcZy34FeClhngHrqSfQk6zx5IT5vwX+I3r8YmJ4EVrb3sC6zyK8IRwUPX8NuLqB5X8O/CR63FCgN7WO3wPfjB43Fug/BH6VMK8DsAE4LXpeBkxNmH8H8PMGtv1X4Pbo8aVReHaMnn+vct/Xek0W8BkwrI55yQR6U/4uLwUW1LNcIfC3hL+NTUB+qv+/4nxTl0vb8n7iEzM73syeiT6WfgzcSmh11WdDwuOdQLdmLHtkYh0e/vvK6ltJkjUmtS1gTQP1AvwN+Bj4opkdB4wCihNqOcXM/mpmm8xsG/DVOmqpS4N1mNkXzOy1qAvhI2BCkuutXHfV+tx9H2F/9k9YJqnfW9RldgZQeczliWjZyi6io4D36njpYUDneuYloyl/l/XVUFlvnoWzrSYCH7j7682sqV1SoLcttU+Vuwd4B/icu/cAvk9oMR9I6wldDwCYmVEzfGprSY3rCQFQqcHTKqM3l4cJBwMvBZ51980Ji8wDHgeOcveewH1J1lJvHWZ2MPAYcBtwmLv3InQdVK63sdMb1wGDEtbXgbB/y5Ooq7bLou0+Z2YbgJWEoL48mv8+cEwdr9tI6Dapa94OIDuhvo6Erp9ETfm7rK8G3H0n4fdTSPj9/bKu5aR+CvS2rTuwDdhhZkOBr7fCNn8P5JvZF6N/7m8B/Q5QjY8C344OmB0C/J8kXvMwoXX3FcKZL7Vr2eruu8zsZGBqCuo4iBCam4C9ZvYFYFzC/I1A3+hgZX3rnmRmZ0UHDq8nHKN4LcnaEl1GCM+RCbcvET6x9CZ0pU20cCpnRzPra2Z5Hs4AehCYbWaHRweBx0T1vAt0N7Nzo+e3EPrWG9LQ7/xpwkHiq6KDrj3MbHTC/IcJv7vPR/VKEyjQ27brCK2v7YRW0a8P9AbdfSMhJO4AthBaW28Q+mBTXePdhAN1bwMLCC3hxupbCfyTELTP1Jp9JXBbdDbGTYQwbVEd7v4RcC2hu2ArcBHhTa9y/juEVmdpdNbHobXqXUzYP3cT3hQmApPcfU+StQFgZqcRum/ucvcNlbeorlLgS+6+mnDw8v9Etb4OnBCt4lpgKbAwmvcjwNz9Q+BqwptjeTQvsQuoLvX+zt19GzAeuJDwZrccODPhtS8S+s9fc/d6u/KkbhYdgBBpFgtf2FkHXOTuL6W7Hmn7zOxF4H53fzDdtbQ1aqFLk5nZRDPrFZ0P/T3CaYv/THNZEgNRV9hw4DfprqUtUqBLc5wGrCJ0EZwLTHH3+rpcRJJiZkXAH4BvufuOdNfTFqnLRUQkJtRCFxGJibSNkdC3b1/PyclJ1+ZFRNqkhQsXbnb3Ok8VTlug5+TkUFJSkq7Ni4i0SWZW7zem1eUiIhITCnQRkZhQoIuIxERGDRy/Z88eysrK2LVrV7pLkQZ06dKFAQMG0KlTY0N6iEhryqhALysro3v37uTk5BAG8ZNM4+5s2bKFsrIyBg8e3PgLRKTVNNrlYmb3m9kHZvZOPfMtusTUSjNbZGb5zS1m165dHHLIIQrzDGZmHHLIIfoUleGKiiAnBzp0CPdFLb7ctepoE3U0dgUMwoD5+URXrKlj/vmES10ZcDJhlLRG13viiSd6bUuWLNlvmmQm/a4y1yOPuGdnu0P1LTs7TFcdbb8OEi6rWPvWaAvd3V8kDJlZn8nAw9G2/gH0MrMjWvQuI9JGZUJL8OabYefOmtN27gzTVUe860jFWS79qXkJqtqXz6piZjPMrMTMSjZt2pSCTafWli1bGDlyJCNHjuTwww+nf//+Vc93796d1DquuOIKli1b1uAyd911F0Up/E/fuHEjHTt25L777kvZOqXpiopgxgxYsya0v9asCc9bO9TXrm3adNURozrqa7on3ghXY6+vy+X3RBe0jZ7/GShobJ2p6HJ55BH3QYPczcJ9Kj9C3XLLLf6Tn/xkv+n79u3zvXv3pm5DKTBnzhw/7bTT/Oyzz261barLZX+DBtX8OF15GzRIdaiO1NVBS7pcklBOzestNvd6iE3Smq2hlStXkpubS2FhIcOGDWP9+vXMmDGDgoIChg0bxq233lq17Gmnncabb75JRUUFvXr1YubMmeTl5XHKKafwwQcfAPDd736X2bNnVy0/c+ZMRo8ezZAhQ3jllVcA2LFjBxdeeCG5ublcdNFFFBQU8Oabb9ZZX3FxMbNnz2bVqlWsX7++avozzzxDfn4+eXl5TJgwAYDt27dz+eWXM2LECEaMGMGTTz6Z+h3WTmVKS3DWLMjOrjktOztMVx0xr6O+pE+80XAL/fPUPCj6z2TW2dIW+oF+101soa9YscLNzBcsWFA1f8uWLe7uvmfPHj/ttNN88eLF7u4+ZswYf+ONN3zPnj0O+LPPPuvu7tdee63fdttt7u5+8803+5133lm1/A033ODu7k899ZSfe+657u5+2223+Te+8Q13d3/zzTe9Q4cO/sYbb+xX5+rVq33IkCHu7n799df77Nmz3d19/fr1ftRRR3lpaWmNer/zne/4dddd5+7h08bWrVubtX/UQt9fprQE3Q/sp1fVkd46aEkL3cyKgVeBIWZWZmb/Zmb/bmb/Hi3yLOFiByuBe4FvpPD9pl6t3Ro65phjKCgoqHpeXFxMfn4++fn5LF26lCVLluz3moMPPpjzzjsPgBNPPJHS0tI6133BBRfst8zLL7/M1KnhGsZ5eXkMGzasztfOmzePL33pSwBMnTqV4uJiAF599VXGjh3LoEHhgvJ9+vQB4Pnnn+eb3/wmEE5B7N27d9L7QBqWKS1BgMJCKC2FffvCfWFh69egOlq/jka/WOTu0xqZ78A3U1ZRkgYODN0sdU0/ELp27Vr1eMWKFfz0pz/ln//8J7169eKSSy6p87zszp07Vz3OysqioqKiznUfdNBBjS5Tn+LiYjZv3sxDD4UL3K9bt45Vq1Y1aR2SGpX/nDffHBoWAweGME9XeEj702bHcklna+jjjz+me/fu9OjRg/Xr1zN//vyUb2PMmDE8+mi4KP3bb79d5yeAJUuWUFFRQXl5OaWlpZSWlnL99dczb948Tj31VF544QXWRO96W7eGM0/Hjx/PXXfdBYTutg8//DDltbdnmdISlPapzQZ6YSHMnQuDBoFZuJ87t3X+gfLz88nNzeX444/nsssuY8yYMSnfxtVXX015eTm5ubn88Ic/JDc3l549e9ZYpri4mClTptSYduGFF1JcXMxhhx3G3XffzeTJk8nLy6Mw2jG33HILGzduZPjw4YwcOZKXXnop5bWLSHqk7ZqiBQUFXvsCF0uXLmXo0KFpqSfTVFRUUFFRQZcuXVixYgUTJkxgxYoVdOyYGcPv6Hclkh5mttDdC+qalxnpIPv55JNPGDduHBUVFbg799xzT8aEuYhkJiVEhurVqxcLFy5Mdxki0oa02T50ERGpSYEuIhITCnQRkZhQoIuIxIQCPcHYsWP3+5LQ7NmzufLKKxt8Xbdu3YDwLc2LLrqozmXOOussap+mWdvs2bPZmTBg8vnnn89HH32UTOlJGTlyZNVwAiISPwr0BNOmTWPevHk1ps2bN49p0xoc/aDKkUceyWOPPdbs7dcO9GeffZZevXo1e32Jli5dyt69e3nppZfYsWNHStYpIplFgZ7goosu4plnnqm6mEVpaSnr1q3j9NNPrzovPD8/nxNOOIGnnnpqv9eXlpYyfPhwAD799FOmTp3K0KFDmTJlCp9++mnVcldeeWXV0Lu33HILAHPmzGHdunWMHTuWsWPHApCTk8PmzZsBuOOOOxg+fDjDhw+vGnq3tLSUoUOH8rWvfY1hw4YxYcKEGttJVFxczKWXXsqECRNq1L5y5UrOOecc8vLyyM/P57333gPgv/7rvzjhhBPIy8tj5syZLdqvItI6MvY89G9/G+oZ/rvZRo6EKAvr1KdPH0aPHs1zzz3H5MmTmTdvHhdffDFmRpcuXXjiiSfo0aMHmzdv5uSTT2bSpEn1XtD67rvvJjs7m6VLl7Jo0SLy86uvnT1r1iz69OnD3r17GTduHIsWLeKaa67hjjvu4IUXXqBv37411rVw4UIeeOABXnvtNdydk046iTPPPJPevXuzYsUKiouLuffee7n44ot5/PHHueSSS/ar59e//jV/+tOfePfdd/nZz37Gl7/8ZQAKCwuZOXMmU6ZMYdeuXezbt4/nnnuOp556itdee43s7OyqcWBEJLOphV5LYrdLYneLu3PTTTcxYsQIzjnnHMrLy9m4cWO963nxxRergrXyYhKVHn30UfLz8xk1ahSLFy+uc+CtRC+//DJTpkyha9eudOvWjQsuuKBqDJbBgwczcuRIoP4hektKSujbty8DBw5k3LhxvPHGG2zdupXt27dTXl5eNR5Mly5dyM7O5vnnn+eKK64gOxr9rHLoXRHJbBnbQm+oJX0gTZ48mWuvvZbXX3+dnTt3cuKJJwJQVFTEpk2bWLhwIZ06dSInJ6fOIXMbs3r1am6//XYWLFhA7969mT59erPWU6ly6F0Iw+/W1eVSXFzMu+++S05ODhBGi3z88cd1gFQkZtRCr6Vbt26MHTuWr3zlKzUOhm7bto1DDz2UTp061RiWtj5nnHEGv/rVrwB45513WLRoERDCtGvXrvTs2ZONGzfy3HPPVb2me/fubN++fb91nX766Tz55JPs3LmTHTt28MQTT3D66acn9fPs27ePRx99lLfffrtqiN2nnnqK4uJiunfvzoABA6ouQ/fZZ5+xc+dOxo8fzwMPPFB1gFZdLiJtgwK9DtOmTeOtt96qEeiFhYWUlJRwwgkn8PDDD3P88cc3uI4rr7ySTz75hKFDh/L973+/qqWfl5fHqFGjOP744/nyl79cY+jdGTNmMHHixKqDopXy8/OZPn06o0eP5qSTTuKrX/0qo0aNSupneemll+jfvz9HHnlk1bQzzjiDJUuWsH79en75y18yZ84cRowYwamnnsqGDRuYOHEikyZNoqCggJEjR3L77bcntS0RSS8NnyvNot+VSHo0NHyuWugiIjGhQBcRiYmMC/R0dQFJ8vQ7EslMGRXoXbp0YcuWLQqMDObubNmyhS5duqS7FBGpJaPOQx8wYABlZWVs2rQp3aVIA7p06cKAAQPSXYaI1JJRgd6pUycGDx6c7jJERNqkjOpyERGR5lOgi4jEhAJdRCQmFOgiIjGhQBcRiQkFuohITCQV6GY20cyWmdlKM9vvemRmNsjM/mxmi8zsr2amk5RFRFpZo4FuZlnAXcB5QC4wzcxyay12O/Cwu48AbgVuS3Whsr+iIsjJgQ4dwn1RUborEpF0SqaFPhpY6e6r3H03MA+YXGuZXOAv0eMX6pgvKVZUBDNmwJo14B7uZ8xQqIu0Z8kEen/g/YTnZdG0RG8BF0SPpwDdzeyQlpcn9bn5ZoguKFRl584wXUTap1QdFP0P4EwzewM4EygH9tZeyMxmmFmJmZVovJaWWbu2adNFJP6SCfRy4KiE5wOiaVXcfZ27X+Duo4Cbo2kf1V6Ru8919wJ3L+jXr18LypaBA5s2XUTiL5lAXwAca2aDzawzMBV4OnEBM+trZpXruhG4P7VlSm2zZkF2ds1p2dlhuoi0T40GurtXAFcB84GlwKPuvtjMbjWzSdFiZwHLzGw5cBigWDnACgth7lwYNAjMwv3cuWG6iLRPGXWRaBERaZguEi0i0g4o0EVEYkKBLiISEwp0EZGYUKCLiMSEAl1EJCYU6CIiMaFAlxbTML4imaFjuguQtq1yGN/KkR8rh/EFfWtVpLWphS4tomF8RTKHAl1aRMP4imQOBbq0iIbxFckcCnRpEQ3jK5I5FOjSIhrGVyRz6CwXabHCQgW4SCZQC11EJCYU6CIiMaFAFxGJCQW6iEhMKNBFRGJCgS4iEhMKdIkNjfoo7Z3OQ5dY0KiPImqhS0xo1EcRBbrEhEZ9FFGgS0xo1EcRBbrEhEZ9FFGgS0xo1EcRneUiMaJRH6W9UwtdRCQmFOgiIjGRVKCb2UQzW2ZmK81sZh3zB5rZC2b2hpktMrPzU1+qiIg0pNFAN7Ms4C7gPCAXmGZmubUW+y7wqLuPAqYC/y/VhYqISMOSaaGPBla6+yp33w3MAybXWsaBHtHjnsC61JUoIiLJSCbQ+wPvJzwvi6Yl+gFwiZmVAc8CV9e1IjObYWYlZlayadOmZpQrIiL1SdVB0WnAg+4+ADgf+KWZ7bdud5/r7gXuXtCvX78UbVpERCC5QC8Hjkp4PiCalujfgEcB3P1VoAvQNxUFiohIcpIJ9AXAsWY22Mw6Ew56Pl1rmbXAOAAzG0oIdPWpiIi0okYD3d0rgKuA+cBSwtksi83sVjObFC12HfA1M3sLKAamu7sfqKJFRGR/SX31392fJRzsTJz2/YTHS4AxqS1NRESaQt8UFRGJCQW6iEhMKNBFRGJCgS4iEhMKdBGRmFCgi4jEhAJdRCQmFOgiIjGhQBcRiQkFuohITCjQRURiQoEuIhITCnQRkZhQoIuIxIQCXUQkJhToIiIxoUAXEYkJBbqISEwo0EVEYkKBLiISEwp0EZGYUKCLiMSEAl1EJCYU6CIiMaFAFxGJCQW6iEhMKNBFRGJCgS4iEhMKdBGRmFCgi4jERFKBbmYTzWyZma00s5l1zL/TzN6MbsvN7KPUlyoiIg3p2NgCZpYF3AWMB8qABWb2tLsvqVzG3a9NWP5qYNQBqFVERBqQTAt9NLDS3Ve5+25gHjC5geWnAcWpKE5ERJKXTKD3B95PeF4WTduPmQ0CBgN/qWf+DDMrMbOSTZs2NbVWERFpQKoPik4FHnP3vXXNdPe57l7g7gX9+vVL8aZFRNq3ZAK9HDgq4fmAaFpdpqLuFhGRtEgm0BcAx5rZYDPrTAjtp2svZGbHA72BV1NbooiIJKPRQHf3CuAqYD6wFHjU3Reb2a1mNilh0anAPHf3A1OqiIg0pNHTFgHc/Vng2VrTvl/r+Q9SV5aIiDSVvikqIhITCvRmKCqCnBzo0CHcFxWluyIRkSS7XKRaURHMmAE7d4bna9aE5wCFhemrS0RELfQmuvnm6jCvtHNnmC4ikk4K9CZau7Zp00VEWosCvYkGDmzadBGR1qJAb6JZsyA7u+a07OwwXUQknRToTVRYCHPnwqBBYBbu587VAVERST+d5dIMhYUKcBHJPGqhi4jEhAJdRCQmFOgiIjGhQBcRiQkFuohITCjQRURiQoEuIhITCnQRkZhQoIuIxIQCXUQkJhToIiIxoUAXEYkJBbqISEwo0EVEYkKBLiISEwp0EZGYUKCLiMSEAl1EJCYU6CIiMaFAFxGJCQW6iEhMJBXoZjbRzJaZ2Uozm1nPMheb2RIzW2xmv0ptmSIi0piOjS1gZlnAXcB4oAxYYGZPu/uShGWOBW4Exrj7h2Z26IEqWERE6pZMC300sNLdV7n7bmAeMLnWMl8D7nL3DwHc/YPUlikiIo1JJtD7A+8nPC+LpiU6DjjOzP5uZv8ws4l1rcjMZphZiZmVbNq0qXkVi4hInVJ1ULQjcCxwFjANuNfMetVeyN3nunuBuxf069cvRZsWERFILtDLgaMSng+IpiUqA5529z3uvhpYTgh4ERFpJckE+gLgWDMbbGadganA07WWeZLQOsfM+hK6YFalsE4REWlEo4Hu7hXAVcB8YCnwqLsvNrNbzWxStNh8YIuZLQFeAK539y0HqmgREdmfuXtaNlxQUOAlJSVp2baISFtlZgvdvaCuefqmqIhITDT6xSIRaRvcYdMmWLoUliwJ9x98AN27Q48eyd26dYOsrHT/JM3jDp99Bp98Ajt21LyvfHzwwdC3L/TrF269ekGHGDVrFegibYw7lJdXB3dleC9ZAlsSjlx16wZHHhnC7OOPw30yunVLLvwbe6Po3Ln++nftqj90az9uyrS9e5u2L7OyqgM+MegTb4nT+/aFjhmcmhlcmkhyduyAv/0N5s+HV18NYTJgAPTvH+4TH/ft23ZaZPv2QWlpzcCufLx9e/VyffpAbi5ceGG4Hzo03PfvD2bVy+3dWx3uTb1t2FDz+b59jdd/0EHV4d6hQ83wTeb1ievp2jW80VTed+sWfr7a0+p6nHi/c2f4FFN527y55vO33gr3H35Yfz29eyf/BtCvX/hU0FoU6NLmuMOiRSHA//hHeOkl2L0bunSBk08OofGXv8C6dfu32Dp3Dq3W2kGf+PiII1q3FbZnD7z33v7BvWwZfPpp9XJHHBHC+vLLawZ3v341g7s+WVnQs2e4tYR7CMZk3wy2bQuvSSZ0a0/r2hU6dWpZvc1RURE+7TQU/ps2hd/ba6+FeRUVda+ra9f9g376dBg7NvV1t6lALyqCm2+GtWth4ECYNQsKC9NdlbSGDz6AP/0pBPgf/xhajADDh8PVV8OECXD66TVbQ3v3wsaNoXuirCzcEh+XlMCTT4aP/4k6dIDDDms49Pv3b3rLa9cuWL58/+BesSKEeqVBg0JQn312dXAPHRpahpnArDpsjzgi3dUcGB07hr+Bww5Lbnl3+Oijxt8ANm6ExYth/PgDU3ebOW2xqAhmzAgtg0rZ2TB3rkI9jnbvhldeCeE9fz68/nqYfsgh4Z9hwoRw6197VKEmcoetW6uDvr7w37Zt/9f26VN/6PfoEYI6MbxXraruaujQAY45pmZLOzcXhgwJrVOR+jR02mKbCfScHFizZv/pgwaFfkZp29xh5crqAH/hhdDXmpUFp5wC554bbvn56TkL45NPGg/9D+oYY7RTJzjuuOrArgzvY48NXUQiTdVQoLeZLpe1a5s2XTLfxx+Hvu7588Nt9eowffBguOSSEOBnnx1au+nWrVtoPQ8ZUv8yu3eHfvvy8vDx+3Ofg6OPTk8fsLRPbSbQBw6su4U+cGDr1yLNs3dv6DqpDPBXXw3TunYNwX3ddSHEjzkmuYN8maZz5/BJMicn3ZVIe9VmAn3WrLr70GfNSl9N0rjy8upulOefrz5POj8fbrghBPgpp9R/zrKIJK/NBHrlgU+d5ZLZPv00nEZY2QpfvDhMP/xw+Pznw4HM8ePhUF2kUCTl2kygQwhvBXjm2bEDHnwQfve78AWfXbtCi/v00+Gyy0IrfMSIttmNItKWtKlAl8zy8cdw111wxx3hnNshQ0K32Lnnwplnhr5xEWk9CnRpsg8/hDlz4Kc/DY8nToTvfQ9OPTXdlYm0bwp0SdrmzXDnnfDzn4fW+aRJ8N3vwr/8S7orExFQoEsSNmyA//kfuPvucJbRhReGIM/LS3dlIpJIgS71Ki+H//7vMLzC7t0wbRrcdFP4pqOIZJ42F+g7doRwyZSBiuKotBR+/GN44IEw9sill8KNN4avq4tI5mojI0NXu/deOOoouOaaMNiRpM7KlfCVr4Tgvv9+uOKKMMDU/fcrzEXagjYX6OecE/pwf/GLEDIXXRS+Qi7Nt3RpGDtlyBAoLoYrrwxvlr/4hb7GLtKWtLlAHz4cHnooDOR0ww3w5z+H0+VOPRV++9umX4KqPVu0CC6+GIYNgyeegO98J+zXOXPCMLAi0ra0uUCv1L8/3HYbvP9+OB96w4bQch8yJJxWt2NHuivMXCUl8K//Gs5S+cMfQv/4mjXwk5+Er+iLSNvUZgO9UrduoT99xQr4zW/C5Z2uvjr0s990E6xfn+4KM8crr8B554Xzxv/2N/jBD0KQz5oVLo8lIm1bmw/0SllZ1f3pf/97uF7fj38cLoAxfXroXmiP3OGvf4Vx42DMmNA6/9GPQpDfcovOFhKJk9gEehxUsskAAAZgSURBVKJTT4XHHw+t9q9/PbTc8/LCSH/z54eQizv3MGztGWeEN7fFi+H228MpiTfemBkXjRCR1IploFc65hj42c9CP/uPfgRvvx3GHRkxIpxj/dln6a4w9dzh97+Hk08Og2SVloZ9sHp1uICEBswSia9YB3qlPn1Cq7S0NAS5WTjfOicnBP3WremusOX27Qtn+Zx4Inzxi+H6lvfcE84tv+qqpl+hXkTannYR6JUOOij0p7/1Vuh6GTEiXDDjqKNC6K1cme4Km27v3nDu+IgR4SyfTz4Jb1rLl4ehbA86KN0VikhrMU9Th3JBQYGXlJSkZduJ3n47jOddVAQVFeF0vuuuC/3wmXRBhs8+g/feg2XLat7efTcMYZubG96cvvSlcIBYROLJzBa6e0Gd85IJdDObCPwUyALuc/cf15o/HfgJUB5N+rm739fQOjMl0CutXx/OX7/77hCQJ50Ugn3KFOjYSiPeuIc6aof28uWhD3zfvupljzii+ir048eHOju0q89bIu1TiwLdzLKA5cB4oAxYAExz9yUJy0wHCtz9qmSLyrRAr1R5ObU77wwt4pwc+Pa3Q5979+6p2cbOnSGk6wru7durlzv4YDjuuOrgrrwdd5zOUhFpr1oa6KcAP3D3c6PnNwK4+20Jy0wnJoFeae9eePrpcKrfK69Az57hFMhrrgnfUm3Mvn3hYtZ1Bff771cvZxYueF07tIcMCdtRq1tEEjUU6Ml0JvQHEiKIMuCkOpa70MzOILTmr3X39+tYps3IygrdGFOmwD/+ES7wcPvtob992rTQHZOXB9u27R/Yy5aFc+B37apeX48eIaTPPLNmaB97rM5AEZHUSFXv8O+AYnf/zMy+DjwEnF17ITObAcwAGDhwYIo2feCdfHL4ctLq1TB7Nvzv/8Ivfxm+Lr95c/VyWVlw9NHV/dqJwX3YYZl1kFVE4iclXS61ls8Ctrp7z4bWm+ldLg358MMwLvvy5TX7uI8+Gjp3Tnd1IhJnLe1yWQAca2aDCWexTAW+XGsDR7h75TBYk4ClLag34/XuHYbuFRHJJI0GurtXmNlVwHzCaYv3u/tiM7sVKHH3p4FrzGwSUAFsBaYfwJpFRKQO7f6LRSIibUlDXS46KU5EJCYU6CIiMaFAFxGJCQW6iEhMKNBFRGJCgS4iEhNpO23RzDYBa9Ky8dTpC2xudKn2Q/ujmvZFTdofNbVkfwxy9351zUhboMeBmZXUdz5oe6T9UU37oibtj5oO1P5Ql4uISEwo0EVEYkKB3jJz011AhtH+qKZ9UZP2R00HZH+oD11EJCbUQhcRiQkFuohITCjQm8HMjjKzF8xsiZktNrNvpbumdDOzLDN7w8x+n+5a0s3MepnZY2b2rpktja761W6Z2bXR/8k7ZlZsZl3SXVNrMbP7zewDM3snYVofM/uTma2I7nunansK9OapAK5z91zgZOCbZpab5prS7VvE/EpVTfBT4A/ufjyQRzveL2bWH7gGKHD34YSL5ExNb1Wt6kFgYq1pM4E/u/uxwJ+j5ymhQG8Gd1/v7q9Hj7cT/mH7p7eq9DGzAcDngfvSXUu6mVlP4AzgfwHcfbe7f5TeqtKuI3CwmXUEsoF1aa6n1bj7i4SruCWaDDwUPX4I+NdUbU+B3kJmlgOMAl5LbyVpNRu4AdiX7kIywGBgE/BA1AV1n5l1TXdR6eLu5cDtwFpgPbDN3f+Y3qrS7rCEazBvAA5L1YoV6C1gZt2Ax4Fvu/vH6a4nHczsC8AH7r4w3bVkiI5APnC3u48CdpDCj9RtTdQ/PJnwRnck0NXMLklvVZnDw3njKTt3XIHeTGbWiRDmRe7+23TXk0ZjgElmVgrMA842s0fSW1JalQFl7l75ie0xQsC3V+cAq919k7vvAX4LnJrmmtJto5kdARDdf5CqFSvQm8HMjNBHutTd70h3Penk7je6+wB3zyEc7PqLu7fbFpi7bwDeN7Mh0aRxwJI0lpRua4GTzSw7+r8ZRzs+SBx5Grg8enw58FSqVqxAb54xwKWE1uib0e38dBclGeNqoMjMFgEjgR+luZ60iT6pPAa8DrxNyJx2MwyAmRUDrwJDzKzMzP4N+DEw3sxWED7B/Dhl29NX/0VE4kEtdBGRmFCgi4jEhAJdRCQmFOgiIjGhQBcRiQkFuohITCjQRURi4v8DrI6mYWCIPaAAAAAASUVORK5CYII=\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAXoAAAEICAYAAABRSj9aAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADh0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uMy4xLjIsIGh0dHA6Ly9tYXRwbG90bGliLm9yZy8li6FKAAAgAElEQVR4nO3deXyU1dn/8c/FLoLsisoSbK3sa4papIgLxQ1+VGuhcdeH4lJbbfuUurb40FrrY63LY6X94UYK9bG1xQqirQj6syrBIqioILIEUALIGrfA9fvj3IFJmCSTMMlM7nzfr9e8ZuZer0yS6z5zzrnPMXdHRETiq1GmAxARkdqlRC8iEnNK9CIiMadELyISc0r0IiIxp0QvIhJzSvQNjJk1NrNdZtYtndtmkpl92cxqpZ9w+WOb2bNmllcbcZjZzWb2u5ruL1IRJfosFyXa0sdeM/sk4X3ShFMZd9/j7q3cfW06t81WZvYPM7slyfJzzWy9mTWuzvHcfZS756chrtPMbHW5Y9/m7pMO9thVnNPN7Ie1dQ7JTkr0WS5KtK3cvRWwFjgnYdkBCcfMmtR9lFntEeDCJMsvBGa4+546jieTLga2AhfV9Yn1d5lZSvT1nJn9l5n9ycxmmtlO4AIzO9HMXjGzbWa20czuMbOm0fZNolJdTvR+RrR+rpntNLN/mVmP6m4brT/DzN4zs+1mdq+Z/T8zu6SCuFOJ8btmttLMPjazexL2bWxmvzGzLWa2ChhdyUf0F6CzmX0tYf8OwJnAo9H7MWa2xMx2mNlaM7u5ks/7pdKfqao4zOwKM1sefVbvm9kV0fI2wFNAt4RvZ4dHv8uHE/YfZ2ZvRZ/R82Z2XMK6QjO73syWRZ/3TDNrXkncrYFvAlcBvc1sYLn1X49+H9vNbJ2ZXRgtbxn9jGujdQvNrHmybyRRTCdHr6v1dxnt0y/6BrbVzD40s/80s6PNrNjM2iZsNzRar4tHqtxdj3ryAFYDp5Vb9l/A58A5hAv3IcBXgeOBJsAxwHvANdH2TQAHcqL3M4DNQC7QFPgToaRb3W0PB3YCY6N11wNfAJdU8LOkEuPfgDZADqEkelq0/hrgLaAL0AFYGP6UK/zcHgJ+l/D+aqAg4f0pQJ/o8xsQ/YxnR+u+nHhs4KXSn6mqOKLfyTGARef4BOgfrTsNWJ3kd/lw9LoXsCvarylwA/Au0DRaXwi8AnSOzv0ecEUln8Gl0T6NgLnAbxLW9YjOdX702XcEBkbrHgT+CRwJNAZOiuJJFn8hcHIN/y7bAB8B3weaA4cBQ6N1zwL/kXCeexPj1yOF3JHpAPSoxi+r4kT/fBX7/Qj43+h1suSdmATHAG/WYNvLgBcT1hmwkQoSfYoxnpCw/i/Aj6LXCxOTGqF07pUc+2TChaJ59P5V4HuVbH8f8OvodWWJvrpx/B24OnpdVaL/OfDHhHWNgA+Bk6L3hcD4hPV3AfdVcu4XgDuj1xdGSbVJ9P7m0s++3D6Ngc+APknWpZLoq/N3eSGwqILt8oAFCX8bRcDgdP9/xfmhqpt4WJf4xsx6mtnT0dfbHcAUQimtIh8mvC4GWtVg26MS4/DwX1lY0UFSjDGlcwFrKokXYAGwAzjHzL4CDAJmJsRyopm9YGZFZrYduCJJLMlUGoeZnW1mr0ZVEduAUSket/TY+47n7nsJn+fRCduk9HuLqt6+DpS26TwZbVta1dQVeD/JrkcAzSpYl4rq/F1WFENpvAMs9P4aDWxy99drGFODpEQfD+W79D0IvAl82d0PA24hlLBr00ZCFQYAZmaUTUrlHUyMGwmJoVSl3T+ji86jhEbIC4E57r45YZNZwJ+Bru7eBvhDirFUGIeZHQI8AfwSOMLd2xKqIEqPW1U3zA1A94TjNSJ8vutTiKu8i6LzzjWzD4GVhAR+cbR+HfClJPt9RKh+SbZuN9AyIb4mhCqkRNX5u6woBty9mPD7ySP8/h5Ltp1UTIk+nloD24HdZtYL+G4dnPPvwGAzOyf6p/8+0KmWYnwc+EHUUNcB+EkK+zxKKA1eRuiJUz6Wre7+qZmdAIxPQxzNCcm0CNhjZmcDpyas/wjoGDWSVnTsMWZ2ctRg+WNCG8irKcaW6CJCUh2Y8Pg24RtOO0KV3GgLXU6bmFlHMxvgoUfSw8DdZtY5anweFsXzDtDazL4Rvb+VUHdfmcp+57MJjdPXRI29h5nZ0IT1jxJ+d2dF8Uo1KNHH0w8JpbWdhFLUn2r7hO7+ESF53AVsIZTO/k2o4013jA8QGgiXAYsIJeeq4lsJvEZIwE+XW30l8Muod8gNhCR7UHG4+zbgOkK1w1bgPMLFsHT9m4RS6uqoF8rh5eJ9i/D5PEC4WIwGxrj7FynGBoCZnUSoBrrf3T8sfURxrQa+7e4fEBpNfxLF+jrQLzrEdcByYHG07heAufvHwPcIF8310brEqqRkKvydu/t24HTgXMJF8D1gRMK+Cwn186+6e4VVgpKcRQ0cImll4UakDcB57v5ipuOR+s/MFgLT3f3hTMdS36hEL2ljZqPNrG3Un/tmQvfK1zIclsRAVKXWF/jfTMdSHynRSzqdBKwiVDV8Axjn7hVV3YikxMzygWeA77v77kzHUx+p6kZEJOZUohcRibmsHCuiY8eOnpOTk+kwRETqjcWLF29296RdmrMy0efk5FBQUJDpMERE6g0zq/AOcVXdiIjEnBK9iEjMKdGLiMRcVtbRi0jd+uKLLygsLOTTTz/NdChShRYtWtClSxeaNq1qaKH9lOhFhMLCQlq3bk1OTg5h4FHJRu7Oli1bKCwspEePHlXvEIlN1U1+PuTkQKNG4Tn/oKdvFmk4Pv30Uzp06KAkn+XMjA4dOlT7m1csSvT5+TBxIhQXh/dr1oT3AHl5mYtLpD5Rkq8favJ7ikWJ/sYb9yf5UsXFYbmISEMXi0S/dm31lotI9tiyZQsDBw5k4MCBdO7cmaOPPnrf+88//zylY1x66aW8++67lW5z//33k5+mOt2TTjqJJUuWpOVYdaHKRG9mXc1svpm9bWZvmdn3k2xjZnaPma00s6VmNjhh3cVmtiJ6XFx+33ToVsFEchUtF5GDk842sQ4dOrBkyRKWLFnCpEmTuO666/a9b9asGRAaIffu3VvhMR566CGOO+64Ss9z9dVXk9dA63JTKdGXAD90997ACcDVZta73DZnAMdGj4mEWXEws/aEKcaOB4YCt0ZTl6XV1KnQsmXZZS1bhuUikl6lbWJr1oD7/jaxdHeAWLlyJb179yYvL48+ffqwceNGJk6cSG5uLn369GHKlCn7ti0tYZeUlNC2bVsmT57MgAEDOPHEE9m0aRMAN910E3ffffe+7SdPnszQoUM57rjjePnllwHYvXs35557Lr179+a8884jNze3ypL7jBkz6NevH3379uWGG24AoKSkhAsvvHDf8nvuuQeA3/zmN/Tu3Zv+/ftzwQUXpPcDq0SVjbHuvpEwCTLuvtPMlhMmfX47YbOxwKPRJMyvRJNPHAmcDDzn7lsBzOw5wpRoM9P5Q5RepG+8MVTXdOsWknwDvXiL1KrK2sTS/T/3zjvv8Oijj5KbmwvA7bffTvv27SkpKWHkyJGcd9559O5dtty5fft2RowYwe23387111/P9OnTmTx58gHHdndee+01Zs+ezZQpU3jmmWe499576dy5M3/+85954403GDx48AH7JSosLOSmm26ioKCANm3acNppp/H3v/+dTp06sXnzZpYtWwbAtm3bALjjjjtYs2YNzZo127esLlSrjt7McoBBHDhB8dGEWdxLFUbLKlqe7NgTzazAzAqKioqqExYQ/sBWr4a9e8OzkrxI7ajLNrEvfelL+5I8wMyZMxk8eDCDBw9m+fLlvP322wfsc8ghh3DGGWcAMGTIEFavXp302N/85jcP2Oall15i/PgwN/yAAQPo06dPpfG9+uqrnHLKKXTs2JGmTZvyne98h4ULF/LlL3+Zd999l2uvvZZ58+bRpk0bAPr06cMFF1xAfn5+tW54OlgpJ3oza0WYzPgH7r4j3YG4+zR3z3X33E6dko60KSJZoC7bxA499NB9r1esWMFvf/tbnn/+eZYuXcro0aOT9icvrdcHaNy4MSUlJUmP3bx58yq3qakOHTqwdOlShg8fzv333893v/tdAObNm8ekSZNYtGgRQ4cOZc+ePWk9b0VSSvRm1pSQ5PPd/S9JNlkPdE143yVaVtFyEamnMtUmtmPHDlq3bs1hhx3Gxo0bmTdvXtrPMWzYMB5//HEAli1blvQbQ6Ljjz+e+fPns2XLFkpKSpg1axYjRoygqKgId+db3/oWU6ZM4fXXX2fPnj0UFhZyyimncMcdd7B582aKy9eB1ZIq6+gt9M7/v8Byd7+rgs1mA9eY2SxCw+t2d99oZvOAXyQ0wI4CfpqGuEUkQzLVJjZ48GB69+5Nz5496d69O8OGDUv7Ob73ve9x0UUX0bt3732P0mqXZLp06cJtt93GySefjLtzzjnncNZZZ/H6669z+eWX4+6YGb/61a8oKSnhO9/5Djt37mTv3r386Ec/onXr1mn/GZKpcs5YMzsJeBFYBpT2b7oB6Abg7r+LLgb3ERpai4FL3b0g2v+yaHuAqe7+UFVB5ebmuiYeEak7y5cvp1evXpkOI+NKSkooKSmhRYsWrFixglGjRrFixQqaNMmuQQSS/b7MbLG75ybbPpVeNy8Bld5zG/W2ubqCddOB6VWdR0Qk03bt2sWpp55KSUkJ7s6DDz6YdUm+Jur/TyAikiZt27Zl8eLFmQ4j7WIxBIKIiFRMiV5EJOaU6EVEYk6JXkQk5pToRSTjRo4cecANUHfffTdXXnllpfu1atUKgA0bNnDeeecl3ebkk0+mqu7ad999d5mbl84888y0jEXzs5/9jDvvvPOgj3OwlOhFJOMmTJjArFmzyiybNWsWEyZMSGn/o446iieeeKLG5y+f6OfMmUPbtm1rfLxso0QvIhl33nnn8fTTT++baGT16tVs2LCB4cOH7+vbPnjwYPr168ff/va3A/ZfvXo1ffv2BeCTTz5h/Pjx9OrVi3HjxvHJJ5/s2+7KK6/cN8zxrbfeCsA999zDhg0bGDlyJCNHjgQgJyeHzZs3A3DXXXfRt29f+vbtu2+Y49WrV9OrVy/+4z/+gz59+jBq1Kgy50lmyZIlnHDCCfTv359x48bx8ccf7zt/6dDFpQOqLViwYN/kK4MGDWLnzp01/mxB/ehFpJwf/ADSPXnSwIEQ5cik2rdvz9ChQ5k7dy5jx45l1qxZnH/++ZgZLVq04Mknn+Swww5j8+bNnHDCCYwZM6bCuVMfeOABWrZsyfLly1m6dGmZoYanTp1K+/bt2bNnD6eeeipLly7l2muv5a677mL+/Pl07NixzLEWL17MQw89xKuvvoq7c/zxxzNixAjatWvHihUrmDlzJr///e85//zz+fOf/1zpGPMXXXQR9957LyNGjOCWW27h5z//OXfffTe33347H3zwAc2bN99XXXTnnXdy//33M2zYMHbt2kWLFi2q8WkfSCV6EckKidU3idU27s4NN9xA//79Oe2001i/fj0fffRRhcdZuHDhvoTbv39/+vfvv2/d448/zuDBgxk0aBBvvfVWlYOWvfTSS4wbN45DDz2UVq1a8c1vfpMXX3wRgB49ejBw4ECg8uGQIYyRv23bNkaMGAHAxRdfzMKFC/fFmJeXx4wZM/bdhTts2DCuv/567rnnHrZt23bQd+eqRC8iZVRW8q5NY8eO5brrruP111+nuLiYIUOGAJCfn09RURGLFy+madOm5OTkJB2euCoffPABd955J4sWLaJdu3ZccsklNTpOqdJhjiEMdVxV1U1Fnn76aRYuXMhTTz3F1KlTWbZsGZMnT+ass85izpw5DBs2jHnz5tGzZ88ax6oSvYhkhVatWjFy5Eguu+yyMo2w27dv5/DDD6dp06bMnz+fNWvWVHqcr3/96/zxj38E4M0332Tp0qVAGOb40EMPpU2bNnz00UfMnTt33z6tW7dOWg8+fPhw/vrXv1JcXMzu3bt58sknGT58eLV/tjZt2tCuXbt93wYee+wxRowYwd69e1m3bh0jR47kV7/6Fdu3b2fXrl28//779OvXj5/85Cd89atf5Z133qn2OROpRC8iWWPChAmMGzeuTA+cvLw8zjnnHPr160dubm6VJdsrr7ySSy+9lF69etGrV6993wwGDBjAoEGD6NmzJ127di0zzPHEiRMZPXo0Rx11FPPnz9+3fPDgwVxyySUMHToUgCuuuIJBgwZVWk1TkUceeYRJkyZRXFzMMcccw0MPPcSePXu44IIL2L59O+7OtddeS9u2bbn55puZP38+jRo1ok+fPvtmzKqpKocpzgQNUyxStzRMcf1S3WGKVXUjIhJzqcwwNR04G9jk7n2TrP8xUDq3TBOgF9DJ3bea2WpgJ7AHKKnoaiMiIrUnlRL9w4SZo5Jy91+7+0B3H0iYJnCBu29N2GRktF5JXiSLZWM1rhyoJr+nKhO9uy8Etla1XWQCMLPaUYhIRrVo0YItW7Yo2Wc5d2fLli3VvoEqbb1uzKwloeR/TWJcwLNm5sCD7j6tkv0nAhMBunXrlq6wRCQFXbp0obCwkKKiokyHIlVo0aIFXbp0qdY+6exeeQ7w/8pV25zk7uvN7HDgOTN7J/qGcIDoIjANQq+bNMYlIlVo2rQpPXr0yHQYUkvS2etmPOWqbdx9ffS8CXgSGJrG84mISArSkujNrA0wAvhbwrJDzax16WtgFPBmOs4nIiKpS6V75UzgZKCjmRUCtwJNAdz9d9Fm44Bn3X13wq5HAE9GI8w1Af7o7s+kL3QREUlFlYne3asc+d/dHyZ0w0xctgoYUNPAREQkPXRnrIhIzCnRi4jEnBK9iEjMKdGLiMScEr2ISMwp0YuIxJwSvYhIzCnRi4jEnBK9iEjMKdGLiMScEr2ISMwp0YuIxJwSvYhIzCnRi4jEnBK9iEjMVZnozWy6mW0ys6SzQ5nZyWa23cyWRI9bEtaNNrN3zWylmU1OZ+AiIpKaVEr0DwOjq9jmRXcfGD2mAJhZY+B+4AygNzDBzHofTLAiIlJ9VSZ6d18IbK3BsYcCK919lbt/DswCxtbgOCIichDSVUd/opm9YWZzzaxPtOxoYF3CNoXRsqTMbKKZFZhZQVFRUZrCEhGRdCT614Hu7j4AuBf4a00O4u7T3D3X3XM7deqUhrBERATSkOjdfYe774pezwGamllHYD3QNWHTLtEyERGpQwed6M2ss5lZ9HpodMwtwCLgWDPrYWbNgPHA7IM9n4iIVE+TqjYws5nAyUBHMysEbgWaArj774DzgCvNrAT4BBjv7g6UmNk1wDygMTDd3d+qlZ9CREQqZCEnZ5fc3FwvKCjIdBgiIvWGmS1299xk63RnrIhIzCnRi4jEnBK9iEjMKdGLiMScEr2ISMwp0YuIxJwSvYhIzCnRi4jEnBK9iEjMKdGLiMScEr2ISMwp0YuIxJwSvYhIzCnRi4jEnBK9iEjMKdGLiMRclYnezKab2SYze7OC9XlmttTMlpnZy2Y2IGHd6mj5EjPTTCIiIhmQSon+YWB0Jes/AEa4ez/gNmBaufUj3X1gRTOfiIhI7apyzlh3X2hmOZWsfznh7StAl4MPS0RE0iXddfSXA3MT3jvwrJktNrOJle1oZhPNrMDMCoqKitIclohIw1VliT5VZjaSkOhPSlh8kruvN7PDgefM7B13X5hsf3efRlTtk5ubm30zlouI1FNpKdGbWX/gD8BYd99Sutzd10fPm4AngaHpOJ+IiKTuoBO9mXUD/gJc6O7vJSw/1Mxal74GRgFJe+6IiEjtqbLqxsxmAicDHc2sELgVaArg7r8DbgE6AP9jZgAlUQ+bI4Ano2VNgD+6+zO18DOIiEglUul1M6GK9VcAVyRZvgoYcOAeIiJSl3RnrIhIzCnRi4jEnBK9iEjMKdHH0N698OCDMGIEvP12pqMRkUxToo+ZZcvgpJNg0iR4+WU4/XT44INMRyUimaREHxPFxfDTn8LgwfDee/DII7B4MXzySUj2GzdmOkIRyRQl+hh45hno2xduvx0uvBDeeQcuugj694e5c+HDD0Oy37Kl6mOJSPwo0ddjGzfC+PFwxhnQvDm88AJMnw4dO+7f5vjjYfZsWLkSzjwTdu7MWLgikiFK9PXQ3r3wu99Br17w17/ClCmwZElofE3mlFPg8cdDVc6YMaE6R0QaDiX6embZMhg2DK68EoYMgaVL4eabQ4m+MmPGhHr7BQvg29+GL76om3hFJPOU6OuJ4mKYPDk0tq5cCY8+Cv/4B3zlK6kfIy8P7rsPnnoKLrkkfDMQkfhL23j0UnvmzoWrroLVq+Gyy+COO6BDh5od66qrYMeO0EPnsMPgf/4HwrhzIhJXSvRZbONG+MEPQv16z56hsbWievjqmDwZtm2DX/0K2rQJvXVEJL6U6LNQ6Z2tkyfDZ5/BbbfBj39cdT18dfzyl7B9e0j2bduGc4lIPCnRZ5mlS+G734VXXoFTT4UHHoBjj03/eczg/vv3V+O0aRMaeEUkflJqjDWz6Wa2ycySzhBlwT1mttLMlprZ4IR1F5vZiuhxcboCj5vdu+EnP9nf2PrYY/Dcc7WT5Es1agQPPwznnANXXw0zZtTeuUQkc1LtdfMwMLqS9WcAx0aPicADAGbWnjAj1fGE+WJvNbN2NQ02rubMCXe23nFH6A3zzjtwwQV100jatGloAzj55HDu2bNr/5wiUrdSSvTuvhDYWskmY4FHPXgFaGtmRwLfAJ5z963u/jHwHJVfMBqUDRvg/PPhrLPgkENg4UL4wx9q3qOmplq0gL/9LfTLP/98eP75uj2/iNSudPWjPxpYl/C+MFpW0fIDmNlEMysws4KioqI0hZWd9uwJ3Rp79Qol6P/6r3Bn6/DhmYupdevQjfPYY8PNVa++mrlYRCS9suaGKXef5u657p7bqVOnTIdTa954I9zZevXV8NWvwptvwo03QrNmmY4M2reHZ5+FI44I4+csW5bpiEQkHdKV6NcDXRPed4mWVbS8wdm9O3SRHDIEVq0KDZ/PPQdf/nKmIyvryCPDHbctW4YRL1euzHREInKw0pXoZwMXRb1vTgC2u/tGYB4wyszaRY2wo6JlDcrTT0OfPnDnnXDppaGxNS8ve+9I7dEjXIRKSuC006CwMNMRicjBSLV75UzgX8BxZlZoZpeb2SQzmxRtMgdYBawEfg9cBeDuW4HbgEXRY0q0rEHYsAG+9S04++xQQl64EH7/+1BFku169YJ58+Djj0PJPubNJiKxZu6e6RgOkJub6wUFBZkOo8b27AnDCN9wQ7iz9eabQ7VNNtTDV9eLL8KoUSHxz58fbqwSkexjZovdPTfZuqxpjI2LN96Ar30NrrkmTPqRTY2tNTF8OPzlL6Fh9uyzwyiaIlK/KNGnSXFxuLN1yJAwGfeMGaHqI9saW2vijDMgPz9MNn7uufD555mOSESqQ4k+DebNO/DO1mxubK2J88+HadPC/LR5eaF6SkTqByX6g7BpU0h6o0eHqpkXXgh3ttaHxtaauPxy+O//hieegIkTIQubd0TqrV27QtVvbdDolTXgHibh/vGPQ//4W28NI0CmcxjhbHX99WEs+9tuCw2z//3f8frmIlJXduyAl14K03u+8EKY07lDB/jww/T/TynRV9O774ZhhBcsCA2VDz4YeqQ0JD//eRjL/je/CWPZ33JLpiMSyX4ffxx6sS1YEB7//neYe6JpUxg6NLTxjRgRCpJK9Bny2WdhJqZf/CL0if/978O0fo0aYOWXWUjyO3aEbzNt2sD3v5/pqESyy5Yt4d6Z0sT+xhshiTdvHnrk3XRTSOwnnBBySm1Sok/Biy+GOul33oHx40OS69w501FlVqNG4WK3Y0eY7vCww8JdvyIN1aZNIbG/8EJI7G9Gs3cccgiceCL87GchsR9/fBgxti4p0Vfi44/hP/8zNLDm5ITRHUdrkOV9mjSBP/4xTFxyxRUh2Z97bqajkmy3dy+sWAEFBaHw1KEDdOmy/9G5c/jbynYbN+4vrS9YAMuXh+UtW4aBC8ePD/M8fPWrmb+Pph58nHXPHf70p1AdsWUL/OhH4Wp86KGZjiz7NG8OTz4Z7p6dMAGeegq+8Y1MR5W9Nm4Mz507N4xGbHd4//2Q1AsKQoPj4sWwc2fF+zRqFAbX69q17AUg8XHUUaFuuy4VFpZN7O+9F5a3ahXa6y6+OJTYhwyp+9iqoiEQyvngA7jqqtBfPDc3VE8MHJiRUOqVbdtg5MjQWP3ss3DSSZmOKDt88QX8619hFrE5c/YP/dymDfTuHRryE5+7dau/7T7usHr1gUl927awvlmz8L80ZEj438rNDT/3zp0hiRYWwrp1+18nLtu9u+y5zMLFMtlFoPQCcdRRB9cTbs2a/T1iFiwIo85C+N0NHx6S+ogRMGhQdnwDqWwIBCX6SElJqHu/9VZo3DhMBnLNNeG1pGbTpvAP8OGH4Z9j0KBMR5QZGzeGgsKcOWEU0O3bQyIYPjzcZXzIIfD22+Gr/ttvh8+tVMuW0LPngReAL30pO5JJKfeQgEuTemli3xoNWdi0KfTvH5J5aWLv06dmVRjuoS0o2UUg8bF9+4H7Hn548otA6ePoo8Pvwz0U8kqT+oIFIdEDtGsHX//6/sQ+YEB25gUl+iosWhQaW5csCbMr3Xdf+IOQ6lu3LpTmi4tDI3bPnpmOqPbt2QOvvba/1P7662H5UUeFxH7mmWG458MOS77/li37k37i87qEudmaNoWvfOXAbwFf+UrtN+y5h5FYyyf10hFNGzeGfv3KJvV+/er+vpLEbwbJHuvWhXa38jp0CJ/vhx+G9x077k/qI0aEu97rw7csJfoK7NwZujjdd1+YVem++2DcuIZRd1qb3nsvlF6bNQs3hHTvnumI0q+oKAx9MWdOeN66NSSDr30tJPYzzwwl2oP5W9q5MzRWlr8ArFoVGjQhnPOYYw68APTsGaaHrImNG0MiT0zsH320/3x9+pRN6v37h1JxfbB7N6xff+BFYNeu0M1xxIjwGdbHHKBEnwG91WkAAA4pSURBVMTs2WE6v/Xr4corQ/94DcGbPm+8EXocdOwYSvb1vTvq3r2hpF5aan/ttVDSPfzw/aX2008PX/Nr26efhotp+QvAe++FNoFSXbseeAHo1avs5PObNh2Y1DdsCOvMwval9elDhoQ69tru8y01c9CJ3sxGA78FGgN/cPfby63/DTAyetsSONzd20br9gCls4+udfcxVZ2vNhP9+vVw7bVh6N2+fcNAXSeemL7j5+eHYYnXrg0Na1OnhvFwGqJ//StUWbRrF+7869EjlD579AiPnJzsLgl+/HFoWJ4zJ9S5b9oUkt/QoftL7YMHZ8/X+i++CKX98heA5cvhk0/2b3f44WES+LVry1YPHXfc/qSemxuSeqtWdf9zSM0cVKI3s8bAe8DpQCFhpqgJ7v52Bdt/Dxjk7pdF73e5e7X+XGoj0e/dGyYDmTw5/EPcckvoNpnOblD5+aGuP3HM9pYtw8WkoSb7hQvDqJ6rVoXGrk8/Lbv+yCP3J/7Ei8Axx4SGsrps9HIP30Tmzg3J/eWXw99N+/bh/okzzwzdSOvb3PV794aknngBWLEifL6lSX3QoIrbEKR+ONhEfyLwM3f/RvT+pwDu/ssKtn8ZuNXdn4veZzzRL1sWEvArr4QS5gMP1M448Tk5+1vqE3XvHrqdNXTuoa63NOl/8EHZ1+vW7a97hnAR7tbtwAtA6esOHQ6+LnXHjjAZ+pw5IcGXVlsMHry/1D50aHb2shBJVFmiT6XD1tFAwhc8CoHjKzhRd6AH8HzC4hZmVgCUALe7+19TijoNPvkkjLL461+Hwbcee6x2x4lfu7Z6yxua0r7PnTuHRsvyPv88JPvyF4BVq0JV2+bNZbdv3bribwM5Ocnrkt1DibY0sb/4Yuha26ZNKK2feWYovdf3NgWRROnumTseeMLdE6el6O7u683sGOB5M1vm7u+X39HMJgITAbp163bQgfzjHzBpUrgr75JLQrLv2PGgD1upbt2Sl+jT8OM0CM2ahf7iX/pS8vU7d+5P/okXg5UrQ116Yj00hJ5UiReALVtCgi+98PbvH6rvzjgjtNNk292MIumSSqJfDyT2Ku8SLUtmPHB14gJ3Xx89rzKzF4BBwAGJ3t2nAdMgVN2kEFdSRUVhzPQZM0KD0z//CaecUtOjVc/Uqcnr6KdOrZvzx13r1iE59+9/4Dr30Fia7NvAyy+HIS0OOSRU3d10U0juXbrU/c8gkgmpJPpFwLFm1oOQ4McD3ym/kZn1BNoB/0pY1g4odvfPzKwjMAy4Ix2Bl+cOjzwCP/zh/v7xN95Yt6PElTa4qtdN3TMLJfgjjgj9ocsr7XaoUrs0RFUmencvMbNrgHmE7pXT3f0tM5sCFLj77GjT8cAsL9u62wt40Mz2EqYtvL2i3joHa9u2MONTz56hl0ufPrVxlqrl5SmxZyMleGnIYnXD1IoVoX43W/o1i4jUlYPtdVNvHHtspiMQEck+KvuKiMScEr2ISMwp0YuIxJwSvYhIzCnRi4jEnBK9iEjMKdGLiMScEr2ISMwp0YuIxJwSvYhIzCnRi4jEnBK9iEjMKdGLiMScEr2ISMwp0YuIxFxKid7MRpvZu2a20swmJ1l/iZkVmdmS6HFFwrqLzWxF9Lg4ncGLiEjVqpx4xMwaA/cDpwOFwCIzm51kSsA/ufs15fZtD9wK5AIOLI72/Tgt0YuISJVSKdEPBVa6+yp3/xyYBYxN8fjfAJ5z961Rcn8OGF2zUEVEpCZSSfRHA+sS3hdGy8o718yWmtkTZta1mvtiZhPNrMDMCoqKilIIS0REUpGuxtingBx3708otT9S3QO4+zR3z3X33E6dOqUpLBERSSXRrwe6JrzvEi3bx923uPtn0ds/AENS3VdERGpXKol+EXCsmfUws2bAeGB24gZmdmTC2zHA8uj1PGCUmbUzs3bAqGiZiIjUkSp73bh7iZldQ0jQjYHp7v6WmU0BCtx9NnCtmY0BSoCtwCXRvlvN7DbCxQJgirtvrYWfQ0REKmDunukYDpCbm+sFBQWZDkNEpN4ws8Xunptsne6MFRGJOSV6EZGYU6IXEYk5JXoRkZhTohcRiTkleom9/HzIyYFGjcJzfn6mIxKpW1X2oxepz/LzYeJEKC4O79esCe8B8vIyF5dIXVKJXmLtxhv3J/lSxcVhuUhDoUQvsbZ2bfWWi8SREr3EWrdu1VsuEkdK9BJrU6dCy5Zll7VsGZaLNBRK9DGkXib75eXBtGnQvTuYhedp09QQKw2LEn3MlPYyWbMG3Pf3MslEss+WC05eHqxeDXv3hmcleWlolOhjJlt6mWTTBUekoVOij5ls6WWSLRccEUkx0ZvZaDN718xWmtnkJOuvN7O3o8nB/2lm3RPW7TGzJdFjdvl9Jb2ypZdJtlxwRCSFRG9mjYH7gTOA3sAEM+tdbrN/A7nR5OBPAHckrPvE3QdGjzFpilsqkC29TLLlgiMiqZXohwIr3X2Vu38OzALGJm7g7vPdvfSL+iuEScAlA7Kll0m2XHBEJLVEfzSwLuF9YbSsIpcDcxPetzCzAjN7xcz+T0U7mdnEaLuCoqKiFMKSimRDL5NsueCISJoHNTOzC4BcYETC4u7uvt7MjgGeN7Nl7v5++X3dfRowDcKcsemMSzIjL0+JXSQbpFKiXw90TXjfJVpWhpmdBtwIjHH3z0qXu/v66HkV8AIw6CDiFRGRakol0S8CjjWzHmbWDBgPlOk9Y2aDgAcJSX5TwvJ2ZtY8et0RGAa8na7gRUSkalVW3bh7iZldA8wDGgPT3f0tM5sCFLj7bODXQCvgf80MYG3Uw6YX8KCZ7SVcVG53dyV6EZE6ZO7ZVx2em5vrBQUFmQ5DRKTeMLPF7p6bbJ3ujBURiTklehGRmFOiFxGJOSV6EZGYU6IXEYk5JXoRkZhTohcRiTklehGRmFOiFxGJOSV6EZGYU6IXqSP5+ZCTA40ahWdNlC51RYlepA7k58PEibBmDbiH54kTM5PsdcFpeJToRerAjTdCcXHZZcXFYXldyqYLjtQdJXqROrB2bfWW15ZsueBI3VKiF6kD3bpVb3ltyZYLDmRHFVI2xFAXlOhF6sDUqdCyZdllLVuG5XUpWy442VCFlA0xJMZSqxccd6/yAYwG3gVWApOTrG8O/Cla/yqQk7Dup9Hyd4FvpHK+IUOGuEjczJjh3r27u1l4njEjMzG0bOkeUlt4tGxZ97F07142htJH9+4NKwb39P1OCDP+Jc2pVc4wZWaNgfeA04FCwhyyEzxhSkAzuwro7+6TzGw8MM7dv21mvYGZwFDgKOAfwFfcfU9l59QMUyK1Jz8/1MmvXRtK8lOnQl5e3cbQqFFIaeWZwd69DScGCCX4NWsOXN69O6xenfpxDnaGqaHASndf5e6fA7OAseW2GQs8Er1+AjjVwuSxY4FZ7v6Zu39AKNkPTT10EUm3vLyQQPbuDc91neQhO6qQsiEGqJt2k1QS/dHAuoT3hdGypNu4ewmwHeiQ4r4AmNlEMysws4KioqLUoheReikb2iyyIQaomwtO1jTGuvs0d89199xOnTplOhwRqUV5eTBtWqieMAvP06bV7beLbIgB6uaC0ySFbdYDXRPed4mWJdum0MyaAG2ALSnuKyINUF5eZqqNsjEGqN12k1RK9IuAY82sh5k1A8YDs8ttMxu4OHp9HvB81Ao8GxhvZs3NrAdwLPBaekIXEYmH2m43qbJE7+4lZnYNMA9oDEx397fMbAqhO89s4P8Cj5nZSmAr4WJAtN3jwNtACXB1VT1uREQkvarsXpkJ6l4pIlI9B9u9UkRE6jElehGRmFOiFxGJuaysozezIiDJTcH1Skdgc6aDyBL6LMrS51GWPo/9Duaz6O7uSW9CyspEHwdmVlBRw0hDo8+iLH0eZenz2K+2PgtV3YiIxJwSvYhIzCnR155pmQ4gi+izKEufR1n6PParlc9CdfQiIjGnEr2ISMwp0YuIxJwSfRqZWVczm29mb5vZW2b2/UzHlA3MrLGZ/dvM/p7pWDLJzNqa2RNm9o6ZLTezEzMdUyaZ2XXR/8mbZjbTzFpkOqa6ZGbTzWyTmb2ZsKy9mT1nZiui53bpOJcSfXqVAD90997ACcDV0by5Dd33geWZDiIL/BZ4xt17AgNowJ+JmR0NXAvkuntfwsi44zMbVZ17GBhdbtlk4J/ufizwz+j9QVOiTyN33+jur0evdxL+kZNOndhQmFkX4CzgD5mOJZPMrA3wdcKQ3rj75+6+LbNRZVwT4JBosqKWwIYMx1On3H0hYVj3RInzbz8C/J90nEuJvpaYWQ4wCHg1s5Fk3N3AfwJ7Mx1IhvUAioCHomqsP5jZoZkOKlPcfT1wJ7AW2Ahsd/dnMxtVVjjC3TdGrz8EjkjHQZXoa4GZtQL+DPzA3XdkOp5MMbOzgU3uvjjTsWSBJsBg4AF3HwTsJk1fy+ujqO55LOECeBRwqJldkNmosks0S19a+r8r0aeZmTUlJPl8d/9LpuPJsGHAGDNbDcwCTjGzGZkNKWMKgUJ3L/2G9wQh8TdUpwEfuHuRu38B/AX4WoZjygYfmdmRANHzpnQcVIk+jczMCHWwy939rkzHk2nu/lN37+LuOYSGtufdvUGW2tz9Q2CdmR0XLTqVMMVmQ7UWOMHMWkb/N6fSgBunEyTOv30x8Ld0HFSJPr2GARcSSq5LoseZmQ5Kssb3gHwzWwoMBH6R4XgyJvpm8wTwOrCMkIsa1FAIZjYT+BdwnJkVmtnlwO3A6Wa2gvCt5/a0nEtDIIiIxJtK9CIiMadELyISc0r0IiIxp0QvIhJzSvQiIjGnRC8iEnNK9CIiMff/AXBPuN+qklqkAAAAAElFTkSuQmCC\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "# 绘制结果\n",
    "import matplotlib.pyplot as plt\n",
    "\n",
    "acc = history.history['acc']\n",
    "val_acc = history.history['val_acc']\n",
    "loss = history.history['loss']\n",
    "val_loss = history.history['val_loss']\n",
    "\n",
    "epochs = range(1, len(acc) + 1)\n",
    "\n",
    "plt.plot(epochs, acc, 'bo', label='Training Acc')\n",
    "plt.plot(epochs, val_acc, 'b', label='Validation Acc')\n",
    "plt.title('Training and Validation Accuracy')\n",
    "plt.legend()\n",
    "\n",
    "plt.figure()\n",
    "\n",
    "plt.plot(epochs, loss, 'bo', label='Training loss')\n",
    "plt.plot(epochs, val_loss, 'b', label='Validation loss')\n",
    "plt.title('Training and Validation Accuracy')\n",
    "plt.legend()\n",
    "\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "  如果不使用预训练的词向量模型进行训练，将词嵌入和模型一起进行训练，得到的验证精度在50%以上。增加训练集数量后，不用预训练的精度将会上升到更好的情况（更适应了当前的应用场景下的词向量）。"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.7.7"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}
